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RELATED APPLICATIONS 
This application is a continuation of U.S. 09/656,666, filed on September 7, 2000, 
entitled "SYSTEM AND METHOD FOR CONTENT RETRIEVAL". The entirety of which 
is incorporated herein by reference. 
10 TECHNICAL FIELD 

The present invention relates generally to computer systems, and more particularly to 
a system and method for retrieving web-based content associated with a particular 
communication. 

BACKGROUND OF THE INVENTION 
Global competition has led to an ever increasing demand for accessing relevant 
information quickly. Prompt access to relevant information can make the difference with 
respect to making money over losing money in the stock market. Demands on the media and 
journalists place a premium on obtaining relevant information before the competition. Other 
industries such as in the high technology sector and consulting fields require individuals in 
those industries to be on top of current events and trends with respect to certain markets. 
Accordingly, there is an ever increasing need for systems and methods which facilitate prompt 
access to relevant information. 

SUMMARY OF THE INVENTION 
The present invention relates to a system and method for retrieving content associated 
with a communication. One aspect of the invention relates to retrieving content associated 
with a program broadcast. A search (over the Internet for example) is conducted based on 
closed caption text associated with the communication, and search results are presented to a 
user while viewing the communication. For example, if the search was conducted over the 
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Internet for web pages or sites relating to the text, the search results may include URLs 
(Unified Resource Locators) associated with the web pages or web sites. The text may be 
acted upon initially so as to generate key words in order to focus the search. For example, a 
natural language search system may be employed to provide a more focused search. 
5 Likewise, any suitable system (e.g., expert system, neural network, web browser search 
engines) may be employed to facilitate structuring the search and content retrieval in 
accordance with the present invention. 

According to one specific aspect of the invention, a user can view a television 
broadcast on a personal computer. The television broadcast includes closed caption text 

10 associated with the broadcast. The present invention is also resident on the PC, and it 

performs a content based search over the Internet in connection with the closed caption text 
associated with the broadcast. The search is conducted dynamically, and the invention 
streams to the user search results based on the closed caption text. The search results may be 
cached if desired. The search results are presented in the form of active URLs, which allow 

1 5 the user to activate any particular URL presented and link to the site during the broadcast. 

The URLs may be presented in a separate window on the PC from the broadcast. Likewise, 
another window may be employed to view a web page associated with a particular URL 
activated by the user. 

Thus, the present invention facilitates a user to dynamically retrieve relevant content- 
20 based information corresponding to a particular communication. The present invention is 
highly advantageous in environments where immediate access to relevant content based 
information is desired such as for example in connection with media, stocks, research, etc. 

Another aspect of the invention relates to retrieving content associated with a 
communication based directly on audio communication (e.g., speech). The audio 
25 communication is input into an audio recognition system that recognizes the audio and 
converts the audio communication into a form (e.g., text or code) that the invention can 
employ to conduct a content-based search related to the audio communication. Search results 
are presented back to a user receiving the audio communication. For example, if a user is 
watching a newscast, the present invention will perform the content-based search based on 
30 audio communication associated with the newscast. The invention will provide the user with 
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search results (e.g., URLs) that correspond to the audio communication. 

In accordance with another aspect of the invention, pattern recognition may be 
employed to conduct a content-based search corresponding to visual indicia (e.g., image(s)) 
associated with a particular communication. Search results (e.g., URLs) corresponding to the 
5 visual indicia are provided back to the user. 

Another aspect of the invention provides for employing a first client computer system 
and at least a second client computer system in connection with displaying and/or storing 
search results. For example, a portable computing system is provided which is operative to 
be employed in connection with a content retrieval system, the content retrieval system 

10 including a communications signal analyzing system which receives a communications signal 
and acting on the signal to generate key search terms associated with the communications 
signal; and a search engine system which performs a content based search over the Internet 
using the key search terms. The portable computing system comprises a first component for 
receiving search results from the content retrieval system; and a second component for 

15 displaying and/or storing at least a portion of the search results.lt is to be appreciated that a 
filtering system may be employed in connection with any of the aforementioned aspects of 
the system so as to streamline search results in order to provide a user with more meaningful 
content-based information corresponding to the communication. For example, the search 
may be limited to particular sources (e.g., specific web sites or libraries). 

20 Furthermore, it is to be understood that some if not all of the herein described aspects 

of the present invention may be employed in combination in order to enhance the content- 
based search. All such combinations are intended to fall within the scope of the hereto 
appended claims. 

The following description and the annexed drawings set forth in detail certain 
25 illustrative aspects of the invention. These aspects are indicative, however, of but a few of 
the various ways in which the principles of the invention may be employed and the present 
invention is intended to include all such aspects and their equivalents. Other advantages and 
novel features of the invention will become apparent from the following detailed description 
of the invention when considered in conjunction with the drawings. 

30 
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BRIEF DESCRIPTION OF THE DRAWINGS 
Fig. 1 is a schematic illustration of one specific user interface in accordance with the 
present invention. 

5 Fig. 2 is a schematic block diagram illustrating a system in accordance with the 

present invention. 

Fig. 3 is a schematic block diagram of a client system in accordance with the present 
invention; 

Fig. 4 is a functional block diagram of an aspect of the present invention relating to 
1 0 closed captioned text. 

Fig. 5 is a functional block diagram of an aspect of the present invention relating to 
audio information. 

Fig. 6 is a functional block diagram of an aspect of the present invention relating to 
image information. 

15 Fig. 7 is a functional block diagram of an aspect of the present invention relating to 

image, audio and closed captioned information. 

Fig. 8 is a block diagram of an exemplary operating environment for a system 
configured in accordance with the present invention. 

20 DETAILED DESCRIPTION OF THE INVENTION 

The present invention is now described with reference to the drawings, wherein like 
reference numerals are used to refer to like elements throughout. In the following 
description, for purposes of explanation, numerous specific details are set forth in order to 
provide a thorough understanding of the present invention. It will be evident, however, to 

25 one skilled in the art that the present invention may be practiced without these specific 

details. In other instances, well-known structures and devices are shown in block diagram 
form in order to facilitate description of the present invention. 

Fig. 1 illustrates a specific application of the present invention with respect to a 
television broadcast over the Internet. A user interface 50 is shown which includes a window 

30 50 corresponding to a news broadcast. The news broadcast window 50 displays a television 
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news broadcaster presenting news to viewers. The broadcast employs closed captioned 
technology which provides for presenting text to the viewers which correspond to the 
utterances of the news broadcaster. The closed captioned text is presented via closed caption 
window 54. According to this specific example, the present invention decodes the closed 
5 captioned text (e.g., via an ATI All-in- wonder tuner system resident on a person computer) 

into ASCII text. News stories may be segmented via "»>" in the closed captioned broadcast 
stream - Cable News Network (CNN) and other news broadcasters commonly employ "»>" 
to indicate a story boundary within the closed captioned stream. Additionally, "»" may be 
employed by the present invention in discriminating between different speakers. 

10 The present invention performs keyword extraction on the ASCII text to generate key 

words and/or terms which are employed by a search engine to perform a search on the 
Internet for content related to the broadcast. It is to be appreciated that the present invention 
is not limited to decoding the closed captioned text into ASCII text, and that other decoded 
formats are contemplated and intended to fall within the scope of the hereto appended claims. 

15 The search results are streamed back to the user and presented via search results window 56, 
which lists unified resource locators (URLs) found by the search engine. The search results 
window 56 includes a scroll bar 62a which allows a user to scroll among the list of URLs and 
select a URL of interest. In the present example, a URL 60 corresponding to "U.S. Says 
Hijackers Backed by Pakistan" is selected, and the content associated with the URL 60 is 

20 presented to the viewer via content window 58. The content associated with the URL 60 is a 
detailed news story relating to the United States believing a terrorist group was supported by 
the Pakistani military. A viewer can select any of the URLs displayed in window 56 and 
view the content associated with the URL sorted by date in window 58. A predetermined 
number of URLs are stored by the present invention and aged URLs are deleted as new URLs 

25 are added to the list. Fig. 1 provides one specific application of the present invention, and it 
is to be appreciated that numerous systems and/or methods for displaying content obtained in 
accordance with the present invention may be employed and all such systems and/or methods 
are intended to fall within the scope of the present invention. 

Turning now to Fig. 2, a functional block diagram of a system 100 according to one 

30 aspect of the present invention is illustrated. A communications signal carrying information 
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(e.g., television broadcast, radio broadcast, video, movie, images) is analyzed by a 
communications signal analyzing system (CSAS) 104. The CSAS 104 decodes the signal 
and generates key search terms which are employed by a search engine 108 to perform a 
search for URLs, websites, or other sources of content related to the information contained 
5 within the signal 102. The search engine 108 conducts the search and outputs results (e.g., 
URL data, websites, databases) to be displayed to a user via a display/storage system 112. 
The CSAS 104 may employ any suitable system or systems for key word/term extraction. 
For example, a neural network, an expert system, NLP dll from NLP group may be 
employed in connection with key word/term extraction in accordance with the present 

10 invention. It is to be appreciated that the CSAS 104, search engine 108 and display/storage 
system 1 12 may be part of a single computing system, or some or all 104, 108 and 112 may 
be part of separate computing systems. Furthermore, it is to be appreciated that the CSAS 
104 and/or the search engine 108 may employ filters to refine the search and/or limit the 
search (e.g., to specific websites) in order to mitigate undesired search results. The 

1 5 display/storage system 112 may comprise a single computing system or a plurality of 

computer systems. A plurality of computer systems (e.g., internet television, and/or portable 
data assistant and/or portable computing system and/or wireless communication device 
and/or any other computing system suitable for displaying and/or storing at least data in 
connection with the present invention) may be employed to enhance displaying and/or 

20 organizing and/or storing the data. For example, a first computer system may be employed 
to display a broadcast and a second computer system be employed to receive and display 
and/or store search results (e.g., URL data) in connection with the present invention. As a 
result a user of the present invention could view a broadcast on a first computer system (e.g., 
an internet television) and have search results provided to and stored in a second computer 

25 system (e.g., portable data assistant), which the user could take to a remote location and 
analyze the search results at a later time, for example. 

Turning now to Fig. 3, by way of example, a system 100a in accordance with the 
present invention is a "WebTV™" system (hereinafter "WebTV") or MicrosoftTV 
(hereinafter "MSTV"), which includes a plurality of client systems 120a, 120b, and 120c. 

30 For purposes of brevity, Fig. 3 illustrates three client systems 120a, 120b, and 120c 
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(hereinafter collectively referred to as "client systems 120"), although it is to be appreciated 
that a greater number of client systems may be employed (as indicated by the ellipsis). 
Briefly stated, each client system 120 employs a standard television set as a device for 
browsing the World Wide Web ("the Web") and is programmed and/or configured to 
5 connect to a conventional network infrastructure, such as the Internet 130. The connection 
between each client system 120a, 120b, 120c and the Internet 130 is made through a 
respective communications link or data connection 140a, 140b, 140c. A communications 
link may be, for example, a standard telephone, ISDN (Integrated Services Digital Network), 
cable modem or other similar communications path. In the example of Fig. 3, 

10 communications links 140a and 140b connect the client systems 120a and 120b, 

respectively, to the Internet 130 through a common communications infrastructure 142, such 
as a modem pool at an Internet service provider ("ISP"). The communications link 140c 
connects the client system 120c to the Internet 130 through another communications 
infrastructure 144 such as, for example, another ISP or a private Internet server. 

15 Each client system 120 may access, via the Internet 130 (generally), WebTV 

Network or MSTV services provided by one or more remote WebTV or MSTV servers 150. 
The WebTV/MSTV Network services are used in conjunction with software running in a 
client system to enable the user to browse the Web, and to make use of the Internet in 
various other known ways. A filtering mechanism also may be programmed at the client 

20 system 120 to refine searches and more efficiently navigate the Web. It is to be appreciated 
that each client system 120 also may communicate with one or more other servers 160, 
which also may provide other Internet-based services, as is known in the art. The servers 
150 and/or 160 may comprise multiple physical and logical devices connected in a 
distributed architecture. Each client system 120 may connect to the servers 150 and 160 

25 either through a direct data connection (not shown) (e.g., telephone, ISDN, cable modem, 

etc.) or through the Internet 130 via its associated communications path. Each client system 
120 also receives audio and/or visual programming content in a predetermined format. 
Examples of signal formats include the Motion Picture Experts Group ("MPEG") family of 
compression/decompression standards, although other data formats also may be utilized. 

30 Examples of systems employing such standard formats that may be implemented in a 
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system, in accordance with the present invention, include Direct Satellite Systems ("DSS"), 
Advanced Television (ATV) systems, Digital Television (DTV) systems, Video-on- 
Demand, MPEG over ATM Networks, High Definition Television (HDTV), cable (e.g., 
DVB-C) etc. 

5 A remote device 1 70, such as a personal computer ("PC"), also is illustrated as being 

connectable to the Internet 130 through a communications infrastructure 180. The 
communications infrastructure 180, for example, is a modem bank associated with another 
ISP. While, for purposes of brevity, a single device 170 is shown in Fig. 3, it is to be 
appreciated that numerous PC's and/or other types of remote devices (both wired and 
10 wireless devices) may be connectable to the Internet for use in accordance with the present 
invention. The remote PC 170 is located remotely from the client systems 120, although it 
may be located in generally the same physical location (e.g., an individual's home) as one of 
the client systems. 

Fig. 4 illustrates an aspect of the present invention relating to a system 180 which 

15 decodes closed captioned data associated with a communication. The system 180 includes an 
Internet TV system 192 which displays a television signal broadcast over the Internet. The 
system includes a closed captioned text system which extracts and displays closed captioned 
information in connection with the broadcast. A closed captioned decoding system 186 
decodes the closed captioned information (CCI) and converts the CCI to a format suitable for 

20 use by a keyword generating system (KWGS) 1 88. For example, the CCI may be converted 
into ASCII text which is acted upon by the KWGS to generate key words and/or key terms to 
be employed by a search engine 190. The search engine 190 performs a search using the 
generated key words and/or key terms and a search results display system (SRDS) 192 
displays the search results (e.g., URLs corresponding to websites containing content related 

25 to the closed captioned information of the broadcast). The SRDS 192 may store all or 
portions of the search results. Furthermore, the SRDS 192 may comprise a plurality of 
computer systems, each computer system performing different or similar functions with 
respect to displaying and storing all or respective portions of the search result data. 

Fig. 5 illustrates an aspect of the present invention relating to a system 200 which 

30 decodes audio communications data associated with a communication. The system 200 
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includes an audio communications system 202 which outputs a signal broadcasting audio. 
The system 200 includes an audio decoding system 204 which decodes the audio information 
and converts the audio information to a format suitable for use by a keyword generating 
system (KWGS) 188. The audio decoding system 204 may include a speech recognition 

5 system, and/or neural network and/or expert system and/or other suitable system to facilitate 
converting the audio information to a suitable format. A search engine 206 performs a search 
using the generated key words and/or key terms generated by the KWGS 188, and a search 
results display system (SRDS) 210 displays the search results (e.g., URLs corresponding to 
websites containing context related to the audio information of the broadcast. The SRDS 210 

10 may store all or portions of the search results. Furthermore, the SRDS 210 may comprise a 
plurality of computer systems, each computer system performing different or similar 
functions with respect to displaying and/or storing all or respective portions of the search 
result data. 

Fig. 6 illustrates an aspect of the present invention relating to a system 220 which 

15 decodes image data associated with a communication. The system 220 includes an image 

system 222 which outputs a signal including image data associated with a communication. 
The system 222 includes an image decoding system 224 which decodes the image 
information and converts the image information to a format suitable for use by a keyword 
generating system (KWGS) 226. The image decoding system 224 may include a pattern 

20 recognition system, and/or colour recognition system and/or neural network and/or expert 
system and/or other suitable system to facilitate converting the image information to a 
suitable format. A search engine 228 performs a search using the generated key words and/or 
key terms generated by the KWGS 226, and a search results display system (SRDS) 230 
displays the search results (e.g., URLs corresponding to websites containing content related 

25 to the image information of the broadcast). The SRDS 230 may store all or portions of the 

search results. Furthermore, the SRDS 230 may comprise a plurality of computer systems, 
each computer system performing different or similar functions with respect to displaying 
and/or storing all or respective portions of the search result data. 

Fig. 7 illustrates an aspect of the present invention relating to a system 250 which 

30 decodes image, audio and closed captioned data associated with a communication. The 
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system 250 includes an image communications system 252 which outputs a signal 
broadcasting image data. The system 250 includes an audio communications system 254 
which outputs a signal broadcasting audio. A CCI system 256 outputs CC data. A decoding 
system 258 decodes the image, audio and CCI information and converts the information to a 
format suitable for use by a keyword generating system (KWGS) 188. The decoding system 
204 may include a speech recognition system, and/or pattern recognition system and/or neural 
network and/or expert system and/or other suitable system to facilitate converting the audio 
information to a suitable format. The combination of using image and/or audio and/or CCI 
data facilitates obtaining desired content-based information via search engine 262. The 
search engine 262 performs a search using the generated key words and/or key terms 
generated by the search term generating system 260, and a search results display system 
(SRDS) 264 displays the search results {e.g., URLs corresponding to websites containing 
content related to the audio information of the broadcast). The SRDS 264 may store all or 
portions of the search results. Furthermore, the SRDS 264 may comprise a plurality of 
computer systems, each computer system performing different or similar functions with 
respect to displaying and/or storing all or respective portions of the search result data.In order 
to provide a context for the various aspects of the invention, Fig. 8 and the following 
discussion are intended to provide a brief, general description of a suitable computing 
environment in which the various aspects of the present invention may be implemented. 
While the invention has been described above in the general context of computer-executable 
instructions of a computer program that runs on a computer and/or computers, those skilled 
in the art will recognize that the invention also may be implemented in combination with 
other program modules Generally, program modules include routines, programs, components, 
data structures, etc. that perform particular tasks and/or implement particular abstract data 
types. Moreover, those skilled in the art will appreciate that the inventive methods may be 
practiced with other computer system configurations, including single-processor or 
multiprocessor computer systems, minicomputers, mainframe computers, as well as personal 
computers, hand-held computing devices, microprocessor-based or programmable consumer 
electronics, or on a "television set top box" (hereinafter "STB") which has processing 
capability similar to a computer, and the like. The illustrated aspects of the invention may 
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also be practiced in distributed computing environments where tasks are performed by remote 
processing devices that are linked through a communications network. However, some, if 
not all aspects of the invention can be practiced on stand-alone computers. In a distributed 
computing environment, program modules may be located in both local and remote memory 
5 storage devices. 

With reference to Fig. 8, an exemplary environment 500 for implementing various 
aspects of the invention includes a client system 502 (e.g., a specially configured computer 
that is part of a WebTV or MSTV system), including a processing unit 504, a system memory 
506, and a system bus 508 that couples various system components including the system 

10 memory to the processing unit 504. The processing unit 504 may be any of various 
commercially available processors. 

The system bus 508 may be any of several types of bus structure including a memory 
bus or memory controller, a peripheral bus, and a local bus using any of a variety of 
conventional bus architectures such as PCI, VESA, MicroChannel, ISA, and EISA, to name a 

15 few. The client system 502 memory includes read only memory (ROM) 510 and random 
access memory (RAM) 512. A basic input/output system (BIOS), containing the basic 
routines that help to transfer information between elements within the client system 502, such 
as during start-up, is stored in ROM 510. 

The client system 502 further includes a hard disk drive 514, a magnetic disk drive 

20 516, e.g., to read from or write to a removable disk 518, and an optical disk drive 520, e.g., 
for reading a CD-ROM disk 522 or to read from or write to other optical media. The hard 
disk drive 514, magnetic disk drive 516, and optical disk drive 520 are connected to the 
system bus 508 by a hard disk drive interface 524, a magnetic disk drive interface 526, and an 
optical drive interface 528, respectively. The drives and their associated computer-readable 

25 media provide nonvolatile storage of data, data structures, computer-executable instructions, 
etc. for the client system 502, including for the storage of broadcast programming in a 
suitable digital format. Although the description of computer-readable media above refers to 
a hard disk, a removable magnetic disk and a CD, it should be appreciated by those skilled in 
the art that other types of media which are readable by a computer, such as magnetic 

30 cassettes, flash memory cards, digital video disks, Bernoulli cartridges, and the like, may also 
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be used in the exemplary operating environment, and further that any such media may contain 
computer-executable instructions for performing the methods of the present invention. 

A number of program modules may be stored in the drives and RAM 512, including 
an operating system 530, one or more application programs 532, other program modules 534, 
5 and program data 536. The operating system 530 in the illustrated computer is, for example, 
the "Microsoft Windows NT" Server operating system, although it is to be appreciated that 
the present invention may be implemented with other operating systems or combinations of 
operating systems. 

A user may enter commands and information into the client system 502 through a 
1 0 keyboard 538 and a pointing device, such as a mouse 540. Other input devices (not shown) 
may include a microphone, an IR remote control, a joystick, a game pad, a satellite dish, a 
scanner, or the like. These and other input devices are often connected to the processing unit 
504 through a serial port interface 542 that is coupled to the system bus 508, but may be 
connected by other interfaces, such as a parallel port, a game port, a universal serial bus 
15 (USB), an IR interface, etc. A monitor 544 or other type of display device {e.g., a television) 
is also connected to the system bus 508 via an interface, such as a video adapter 546. In 
addition to the monitor, a computer typically includes other peripheral output devices (not 
shown), such as speakers, printers etc. 

The client system 502 also includes a tuner/receiver 550, which is connected to the 
20 bus 508 through an appropriate interface 552. As described herein, the tuner/receiver 550 
receives audio and/or visual programming that is broadcast over some transmission medium 
(air, coaxial cable, fiber optic cable, etc). The tuner/receiver 550 demodulates the received 
signal and provides it to the bus 508, so that it may, in turn, be displayed at the monitor 544, 
such as a television. 

25 The client system 502 may operate in a networked environment using logical 

connections to one or more remote computers, such as a remote server computer 560 (e.g., a 
WebTV/MSTV server) or a remote PC or STB. The remote computer 560 may be a 
workstation, a server computer, a router, a peer device or other common network node, and 
typically includes many or all of the elements described relative to the client system 502, 

30 although, for purposes of brevity, only a memory storage device 562 is illustrated in Fig. 8. 
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In addition, the remote computer 560 may be programmed to employ masking in accordance 
with the present invention, relative to one or more devices operatively associated therewith, 
including part of the client system 502. The logical connections depicted in Fig. 8 include a 
local area network (LAN) 564 and a wide area network (WAN) 566. Such networking 
5 environments are commonplace in offices, enterprise-wide computer networks, intranets and 

the Internet. 

When used in a LAN networking environment, the client system 502 is connected to 
the local network 564 through a network interface or adapter 568. When used in a WAN 
networking environment, the client system 502 typically includes a modem 566, or is 

10 connected to a communications server on the LAN, or has other means for establishing 

communications over the WAN 566, such as the Internet. The modem 570, which may be 
internal or external, is connected to the system bus 508 via the serial port interface 542. In a 
networked environment, program modules depicted relative to the client system 502, or 
portions thereof, may be stored in the remote memory storage device 562. It will be 

1 5 appreciated that the network connections shown are exemplary and other means of 

establishing a communications link between the computers may be used. 

In accordance with the practices of persons skilled in the art of computer 
programming, the present invention has been described with reference to acts and symbolic 
representations of operations that are performed by a computer, such as the client system 502 

20 or remote computer 560, unless otherwise indicated. Such acts and operations are sometimes 
referred to as being computer-executed. It will be appreciated that the acts and symbolically 
represented operations include the manipulation by the processing unit 504 of electrical 
signals representing data bits which causes a resulting transformation or reduction of the 
electrical signal representation, and the maintenance of data bits at memory locations in the 

25 memory system (including the system memory 506, hard drive 514, floppy disks 518, CD- 
ROM 522) to thereby reconfigure or otherwise alter the computer system's operation, as well 
as other processing of signals. The memory locations where such data bits are maintained are 
physical locations that have particular electrical, magnetic, or optical properties 
corresponding to the data bits. 
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What has been described above are preferred aspects of the present invention. It is, of course, 
not possible to describe every conceivable combination of components or methodologies for 
purposes of describing the present invention, but one of ordinary skill in the art will 
recognize that many further combinations and permutations of the present invention are 
possible. Accordingly, the present invention is intended to embrace all such alterations, 
combinations, modifications and variations that fall within the spirit and scope of the 
appended claims. Furthermore, to the extent that the term "includes" is used in either the 
detailed description or the claims, such term is intended to be inclusive in a manner similar to 
the term "comprising." 
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